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ABSTRACT 

ECOSET was established to meet the demand from policy 
makers and the teaching community in the European Community (EC) 
Member States for evidence of the effectiveness of information 
technology in various learning situations in general and of various 
kinds of software in particular. The main goals of ECOSET were to 
identify organizations in the Member States that were involved in the 
evaluation of educational software; to inventory available evaluation 
instruments and procedures; and to compare these instruments with 
respect to evaluation techniques, cultural embeddedness , and their 
curricular dependencies. A survey of 19 organizations in 10 Member 
States was conducted to obtain this information. Based on the data 
collected, it was concluded that: (1) future cooperation between 
different countries of the EC has to concentrate on optimization of 
the objectivity and reliability of evaluation methods and procedures; 
(2) the underlying methods for evaluation in the Member States are 
relatively free of curricular and cultural influences and the 
Commission should set up a course on evaluation methods to be 
distributed to interested Member States to encourage unity in 
methodology; and (3) formative evaluation activities of * exemplary* 
organizations should be formalized and documented for use by other EC 
countries. (Contains 4 references.) (ALF) 
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The reason for establishing ECOSET is the considerable demand from policy makers and 
the teaching community for evidence of the effectiveness in various learning situations of 
information technology in general and items of software in particular. Thorough evaluation 
is complex and expensive and much saving of effort could be achieved if Member States 
set up joint evaluation programms, or use the outcomes of each others evaluation 
programmes in their own decision making. The Community can help to ensure that a full 
range of appropriate evaluation tools is available to Member States and that evaluators are 
skilled in their use and in the interpretation of the outcomes. In order to enable such 
cooperation it is at first necessary to collect data on the current state of evaluation activities 
in Europe. This was the main goal of ECOSET. 
ECOSET aimed at: 

1. an inventory of available evaluation injtruments and procedures of application in the 
Member States; 

2. a comparison of these ins'.ruments and procedures with respect to the evaluation 
techniques, their cultural embeddedness and their curricular dependencies. 

Evaluation is seen as 'the planned collection of information so that informed judgements 
can be made about the educational value of the microcomputer and the courseware used 
within the classroom' (Weston, 1987). 



* Drs. L.W.F. Bonekamp is an evaluation specialist at ECC (Educational Computing 
Consortium) in Enschede, Holland. The evaluation department of this company is 
specialized in a wide range of formative and summative evaluation activities regarding 
^ educational software. 
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A first concern in this study was to identify the relevant organizations in the Member 
States involved in evaluation or in the implementation of evaluation procedures and 
instruments regarding educational software. For this identification process we primarily 
relied on the expertise of the Euryclee centre,s. We asked the Euryclee directors of the 
European countries for addresses of institutes or organizations in their country that are 
performing evaluation activities regarding the evaluation of educational software. We 
received addresses of 30 organizations. These organizations are the population of this 
research project. All the 30 organizations were sent a questionnaire and 19 organizations 
responded. After a first analysis of the returned questionnaires four significant actors in the 
Member States have been identified. Their sites were visited for in-depth interviewing, and 
discussion with their staff involved in evaluation procedures has taken place. 

Inyentory of the oyerall context. 

In the ECOSET-project data on evaluation activities of 19 organizations in ten member 
states of the EG are gathered by questionnaire. Key persons in four organisations were 
interviewed about evaluation. The participating member states were Germany, the United 
Kingdom, Belgium, Denmark, France, Luxembourg, the Netherlands, Portugal, Spain and 
Italy. Four organizations are interviewed for more and detailed information about these 
activities. Ten of the nineteen organizations do as well summative as formative evaluations 
and four organizations only perform summative evaluation activities. Five organizations 
are involved in formative evaluation activities only. From the data it is evident that 
national and local government have a significant role as commissioners in the evaluation 
activities of the organizations in the different member states. In almost all of the 
organizations (14) the formative evaluation activities deal with own development projects. 
There is only one organization involved in own development projects whereas the 
evaluation activities are only summative. Three organizations are specialized in information 
dissemination, training and evaluation without any involvement in development activities. 
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Number of organizations (n = 


19) 


- Financed by 






* national government 


1 1 1 1 1 1 1 1 1 i t 1 1 


1 1 14 


* local govemm. 


1 1 1 1 1 15 




* industry and trade 

* others 


1 1 1 1 14 




1 1 1 1 1 15 




- Activities regarding 






educational software 






* development 


1 1 t 1 1 i 1 1 1 1 1 i 1 


1 1 i 


* publications 


1 1 17 


_) 13 


* information dissemen. 


1 1 s i II 


* training 


i 1 1 1 ] 1 i 1 


1 1 115 


* evaluation 


1 1 1 1 i 1 1 1 1 1 1 1 1 


1 1 1 1 1 i i 1Q 


* research 


1 1 1 1 1 1 i 1 


"1 13 


- Expertise evaluators 


t 1 1 1 1 1 i i 




* program technical 


1 1 1 1 1 1 1 19 


* educational 


1 1 i 1 1 1 1 1 1 1 1 1 I 


* content matter 


1 1 i i 1 1 1 1 1 IQ 




- Kind of evaluation 






* only summative 


1 1 1 i i4 




* only formative 


[ 1 I 1 1 15 




* summat. Sc format. 


11 10 





Table 1. Overview of the overall context of the organizations. 



Summative evaluation 

The summative evaluation activities in the member states vary from a critical appraisal of a 
piece of courseware to extensive in-depth testing. Three groups of organizations are 
distinguished. The first group does not use formal instruments. The data indicate that this 
group is more or less on the line of product reviews of educational software, A second 
group (5 organizations) is using formal instruments, but the evalr>ation in real classroom 
situations is not a standard procedure. The third group of organizations uses formal 
instruments and the educational software is tried out in real situations. There are 
indications of differences between group 1 and 2 in the relation between development 
activities and the target group for the results. A remarkable aspect is the difference in 
dissemination of results between group 2 and group 3. Teachers in both groups are an 
important target group for the results, but in group 2 there is only one organization using 
educational or other magazines for the dissemination of the results. In group 3 all the 
organizations use magazines for this purpose. 



Number of organizations (n=14) 


- Evaluation activities 




commissioned by 




* government 


1 i 1 1 1 1 1 1 1 10 


* local authorities 


1 1 1 1 i4 


* industry and trade 


1112 


* others 


i 1 12 


- Involvement of 




publishing companies. 


1 1 1 1 1 1 ) 17 


- Training for evaluators 


1 1 1 1 i { 16 


- Evaluation in classrcom 




situations 


1 1 1 1 1 [ 1 i7 


- Assessment of learner 




effectiveness. 


1 1 { 1 t 1 1 1 


- Use of specific criteria for 




selection of courseware 


1 1 1 i t 1 i 1 1 1 i 10 



Table 2, Overview of some methods and procedures. 



For the summative evaluation activities, formal instruments are used by eleven 
organizations. Checklists are used as instruments by six organizations. They are used 
mostly by teachers or by teachers and subject matter specialists. Seven organizations use 
questionnaires for teachers. Four of them use a combination of questionnaires and 
checklists. At two organizations a quesvionnaivd is also filled in by subject matter 
specialists and at one organization they have a special questionnaire for students. 
Prestructured diaries are used at one organization by teachers and specialists. Nine out of 
fourteen organizations developed 'iieir own instruments, the others use already existing 
instruments. Specific instruments for specific courseware are used by five organizations. 



Formative evaluation 

Fifteen organizations perform formative evaluation activities in the decision as well as in 
the design & development phase. Nine of them are also involved in evaluation activities in 
the dissemination phase. In the decision phase thirteen organizations do comparisons with 
other products and eleven of them are looking in the technical feasibility of products. Ten 
organizations do needs analysis studies. As evaluation activities in the design & 
development phase the evaluations of pre-releases of the software and releases of the 
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documentation are most often mentioned. Other activities are evaluations of prototypes and 
continuing consultancy during the design and technical implementation stages. Products 
most often mentioned as being used as input for activities in this phase are the descriptions 
of the ideas behind the product, global prototypes and product definitions. In the 
dissemination phase three organizations assess learner results as one of their evaluation 
activities. In the group of organizations performing formative evaluation activities two 
elements are remarkable. Organizations with few (<6) evaluation activities hardly train 
their evaluators. Almost all the other organizations give training to in house evaluators, 
subject matter specialists and/or teachers. If the number of evaluation activities gives an 
indication of the role of evaluation in the development process, you might say that those 
organizations also value training for evt^uators as a necessity for good evaluations. The 
organizations with a lot of evaluation activities all use data of own or other observer^. The 
aspect of training may also be connected with this fact. For observations the 'inter 
subjectivity' is an important factor. Inter subjectivity you will get by training your 
observers. 

Concluding comments* 

The summative evaluation activities in the different member states can be placed on a 
continuum with on one side subjective, normative evaluation and on the other side the 
criterion-based approach (Dudley Marling, Owston, Searle, 1988). With a simple 'break- 
down* procedure we made rough distinction in three groups of evaluation methodologies 
used in the European organizations, similar to three prototypical models of types of 
evaluation as proposed by Duchastel (1987). The first group are three organizations that do 
not use formal instruments for their evaluation activities. These organizations are more or 
less on the line of product reviews of educational software. A second group are those 
organizations (5) that use formal instruments but the evaluation in real classroom situations 
is not a standard procedure. That does not mean that teachers are not involved in the 
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evaluation activities. These organizations all use teachers as participants in the evaluation 
activities. This group probably has a more systematic way of evaluation with sets of 
mental categories and product characteristics with which they examine a software product. 
The third group of organizations (6) uses formal instruments and evaluates in real 
classroom situations as part of the standard evaluation procedure. The above mentioned 
division in evaluation activities does not mean that there are a lot of similarities in the 
organizations in one group. There can be for instance a lot of difference in the 
specifications of criteria used in the instruments, the number of teachers evaluating the 
program, the main function of the results etc. 

Formative evaluation activities are standard at those IS organizations that are engaged in 
software development. At two organizations the evaluators are independent subcontractors 
for parts of the evaluation activities in the decision phase and in the design & development 
phase. In the dissemination phase there is only one organization responsible for evaluation 
activities as independent subcontractor. In the decision phase there is one organization and 
in the design «& development phase there are three organizations where the evaluators are 
as well member of the development team and independent subcontractor. Although 
preferable (Sparrow, 1973, Watson, 1987) it is not common in the organizations to have 
independant evaluators. 

The data of the questionnaire do not give insight in the standardization of the different 
evaluation procedures in the development process. The interviews give an indication that 
formative evaluation activities differ per project and that procedures and methods of 
evaluation are planned and developed during the development process, instead of planned 
befordiand. Only one organization in the interviews indicate that the evaluation activities 
during the development process are part of a regularly applied quality assurance plan. 



CONCLUSIONS AND RECOMMENDATIONS 



Some conclusions of the ECOSET-project are: 



Future cooperation between the different countries of the EC have to concentrate on the 
optimizattc of the objectivity and the reliability of the evaluation methods and 
procedures. Although there are some parts in the procedure that have their cultural and 
curricular boundaries, there are other parts where cooperation is possible, for instance 
on the terrain of operational definitions of program characteristics, on test consistency 
and validity and on the development of procedures, tools and methodologies. The basis 
for cooperation should be a consensus about the target group and the purpose of the 
summative evaluation, the questions 'why?' and 'for whom?' (Smith and Keep, 1988). 

* The underlying methods for evaluation in the Member States are relatively free of 
curricular and cultural influences. This implies that exchange of information between the 
Member States can take place in principle wi^^out to much difficulty in terms of 
'education-free' exchanges. From the experiences of the project it is recommended that 
the Commission stimulates setting up a course on evaluation methods to be distributed to 
interested Member States. Through such a course, unity in methodology will evolve, 
encouraging similar approaches towards evaluation resulting in compatibility of 
evaluation outcomes. 

♦ In regard to the formative evaluation activities it seems logical that development projects 
and developing organizations will make use of the expertise and experiences of those 
organizations that perform a wide variety of formative evaluation activities in the 
different phases of the development process. One of the main conditions should be that 
the evaluation activities of those 'exemplary' organizations are formalized methods and 
procedures with good documentation, for instance in a 'quality assurance plan', of the 
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different evaluation activities. It will facilitate the transfer of expertise to other EC 
countries. 
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